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SUMMARY  PAGE 


Objective:  To  determine  how  sonar  and  other  information  is  organized  and  prioritized  by  the 
Submarine  Officer  of  the  Deck  (OOD),  for  the  purposes  of  augmenting  our  understanding  of 
submarine  "expertise,"  for  making  recommendations  for  information  displays  of  submarine 
systems,  and  for  training  submarine  officers. 

Method:  Eighty-three  Naval  Officers  performed  information  sorting  and  ranking  tasks  to 
indicate  similarity  and  relative  importance  among  20  categories  of  information  available  on 
submarines.  These  20  submarine  concepts  were  primary  elements  of  information  the  submarine 
OOD  would  encounter  while  on  watch,  with  the  exception  of  tactics  information.  Each  of  the  20 
concepts  was  printed  on  an  index  card.  Research  participants  made  similarity  judgments  by 
sorting  the  20  cards  into  discrete  groups,  according  to  the  participant's  own  definition  of 
similarity.  The  number  of  piles  and  the  concepts  in  each  pile  were  recorded.  Then,  participants 
placed  the  20  cards  into  a  single  pile,  ranked  or  ordered  according  to  relative  importance. 
Participants  repeated  the  ranking  task  four  times  for  four  differing  operational  scenarios. 
Research  participants  were  Submarine  Officer  Basic  Course  students  (both  pre  and  post  course), 
Submarine  Officer  Advanced  Course  students,  and  Post  Department  Head  submariners. 

Findings:  With  regard  to  information  similarity,  submarine  OODs  organize  information  in  two 
basic  dimensions,  one  defined  by  the  source  of  information  (context/noise  v.  contact/signal),  and 
the  other  defined  by  the  destination  (or  primary  user)  of  the  information  (sonar  v.  CONN).  In 
addition  to  the  two-dimensional  structure  in  the  organization  of  all  20  information  items,  there 
was  also  a  more  finely  grained  grouping  or  "clustering"  of  information.  The  expert  group 
organized  information  into  four  clusters  and  the  groups  with  less  expertise  organized  information 
into  five  clusters.  Further,  the  information  in  these  five  clusters  varied  as  a  function  of 
experience.  With  regard  to  information  prioritization,  the  information  regarded  as  the  most 
important  varied  as  a  function  of  operational  scenario  (i.e.,  littoral  v.  pelagic  water  environment; 
neutral  v.  hostile  contact).  This  change  in  prioritization  between  scenarios  was  most  evident  for 
the  most  experienced  submariner  participants;  little  change  was  observed  for  the  least 
experienced. 

Application:  The  trend  towards  fusion  displays  and  the  importance  of  presenting 
"knowledge"  v.  "information"  gives  special  relevance  to  these  findings.  The  cognitive  model 
developed  from  these  data  and  the  converging  evidence  in  the  literature  serve  as  a  guideline  in 
designing  submarine  displays  that  not  only  present  the  most  appropriate  information,  but  present 
it  in  a  manner  that  is  consistent  with  the  operators'  organization  of  information.  This  alignment  is 
important  in  creating  "user-friendly"  equipment. 

ADMINISTRATIVE  INFORMATION 

This  work  was  conducted  under  Work  Unit  51001  entitled:  "Information  Requirements  and  Information 
Organizations  in  Submarine  Combat  Systems."  The  opinions  or  assertions  contained  herein  are  the  private  ones  of 
the  authors  and  are  not  to  be  construed  as  official  or  reflecting  the  views  of  the  Department  of  the  Navy,  Department 
of  Defense,  of  the  United  States  Government.  This  research  has  been  conducted  in  compliance  with  all  applicable 
Federal  Regulations  governing  the  Protection  of  Human  Subjects  in  Research.  This  Technical  Report  was  approved 
for  publication  on  12  July  2004,  and  designated  as  NSMRL  Technical  Report  TR  #1234. 
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ABSTRACT 


The  submarine  environment  is  alien  to  typical  human  experience.  In  a  world  without  direct 
visual  information,  the  submarine  Officer  of  the  Deck  (OOD)  receives  a  variety  of  diverse 
information  as  inputs  and  makes  operational  decisions  based  on  his  understanding.  Although 
there  have  been  efforts  to  describe  this  understanding,  organization,  and  prioritization 
(Kirschenbaum,  1992;  2001;  Laxar,  Moeller,  &  Rogers,  1983;  1989),  a  validated  model  has  yet 
to  be  described.  The  description  of  a  submarine  OOD's  cognitive  organization  of  information  is 
the  objective  of  this  research.  Using  analyses  of  submariner  knowledge  for  concepts  related  to 
responsibilities  for  the  OOD  task,  this  research  examined  how  submarine  officers  cognitively 
organize  this  information  and  how  experience  may  alter  knowledge  representation  and 
conceptual  importance. 

Eighty-three  Naval  Officers,  some  qualified  and  some  not  qualified  as  OOD,  performed 
information  sorting  and  ranking  tasks  to  indicate  similarity  and  relative  importance  among  20 
categories  of  information  available  on  submarines.  These  20  submarine  concepts  were  primary 
elements  of  information  the  submarine  OOD  would  encounter  while  on  watch,  with  the 
exception  of  tactics  information.  Each  of  the  20  concepts  was  printed  on  an  index  card.  Research 
participants  made  similarity  judgments  by  sorting  the  20  cards  into  discrete  groups,  according  to 
the  participant's  own  definition  of  similarity.  The  number  of  piles  and  the  concepts  in  each  pile 
were  recorded.  Then,  participants  placed  the  20  cards  into  a  single  pile,  ranked  or  ordered 
according  to  relative  importance.  Participants  repeated  the  ranking  task  four  times  for  four 
differing  operational  scenarios.  Research  participants  were  Submarine  Officer  Basic  Course 
students  (both  pre  and  post  course),  Submarine  Officer  Advanced  Course  students,  and  Post 
Department  Head  submariners. 

The  results  show  that  more  experienced  personnel  possess  different  mental  models  than  less 
experienced  personnel.  With  regard  to  information  similarity,  submarine  OODs  organize 
information  in  two  basic  dimensions,  one  defined  by  the  source  of  information  (context/noise  v. 
contact/signal),  and  the  other  defined  by  the  destination  (or  primary  user)  of  the  information 
(sonar  v.  CONN).  In  addition  to  the  two-dimensional  structure  in  the  organization  of  all  20 
information  items,  there  was  also  a  more  finely  grained  grouping  or  "clustering"  of  information. 
The  expert  group  organized  information  into  four  clusters  and  the  groups  with  less  expertise 
organized  information  into  five  clusters.  Further,  the  information  in  these  five  clusters  varied  as  a 
function  of  experience.  With  regard  to  information  prioritization,  the  information  regarded  as  the 
most  important  varied  as  a  function  of  operational  scenario  (i.e.,  littoral  v.  pelagic  water 
environment;  neutral  v.  hostile  contact).  This  change  in  prioritization  between  scenarios  was 
most  evident  for  the  most  experienced  submariner  participants;  little  change  was  observed  for  the 
least  experienced. 

This  knowledge  allows  system  designers  to  develop  command  and  control  displays  that  use  this 
cognitive  organization  to  the  user's  advantage  by  developing  systems  that  are  intuitive  to  the 
user.  Additionally,  the  knowledge  of  these  mental  models  of  experts  could  be  used  as  targets  for 
training  less  experienced  personnel. 
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INTRODUCTION 


The  submarine  presents  an  extreme  and  alien  environment  to  its  human  crew.  In  addition  to  life 
support  challenges,  this  environment  presents  significant  psychological  and  cognitive  challenges. 
Humans  are  adapted  for  direct  path  visual  and  auditory  information  in  an  air-filled  medium; 
therefore,  they  are  poorly  suited  for  the  fluid-filled  underwater  medium  with  limited  or  no  visual 
information  and  indirect  path  auditory  information  that  is  presented  visually.  Our  mental 
representation  or  cognition  of  the  world  is  derived  from  our  perception  of  the  world,  so  the 
difficulties  the  submarine  presents  to  perception  can  be  expected  to  extend  to  cognition. 
Compounding  the  lack  of  typical  perceptual  information,  submarine  technology  presents  a 
wealth  of  atypical  information,  information  not  available  in  everyday  experience  [e.g.,  ESM 
(electronic  surveillance  mast)].  Such  information  is  valuable  to  submarine  operations  so 
operators  need  to  adapt  their  cognition  to  incorporate  this  new  perceptual  ability.  Despite  these 
cognitive  challenges,  people  successfully  operate  submarines  all  over  the  world.  This  research  is 
an  exploration  of  the  mental  model(s)  submarine  officers  use  to  execute  their  mission. 

The  manner  in  which  submariners  cognitively  organize  available  information  is  expected  to  be  as 
unique  as  the  submarine  environment.  The  difficulty  and  vagaries  of  submarining  have  been 
compared  to  weather  prediction  and  medical  diagnosis  (Gray  &  Kirschenbaum,  2000; 
Kirschenbaum,  1992).  For  situational  awareness  (Endsley,  1995),  the  submariner  needs  a  wide 
variety  of  information  available  at  his  watch  station  to  build  his  mental  model  (Johnson-Laird, 
1993).  A  variety  of  visual  displays  that  afford  data  fusion  are  available,  especially  given  recent 
technological  advances,  but  the  Officer  of  the  Deck  (OOD)  remains  the  most  effective  integrator 
of  information  available  in  the  control  room  (Ferren,  2000;  Holland,  1999).  This  determination  is 
made  based  on  the  person's  ability  to  resolve  ambiguities  better  and  execute  with  best  knowledge 
from  incomplete  information.  Given  the  uniqueness  and  limited  size  of  the  submarine 
population,  there  is  a  scarcity  of  human-based  research  in  submariner  cognition  and  his 
organization  of  submarine  information  has  not  been  specified.  Accurate  models  of  submariner 
cognition  are  still  in  development. 

Accurate  models  of  submariner  cognition  are  important  in  designing  displays  on  submarines  for 
optimal  use  and  for  guarding  against  information  overload  (Shobe,  2001,  2002;  Shobe  & 
Severinghaus,  2004).  Organizational  systems  for  information  can  be  inherent  in  the  data,  in  the 
person,  or  in  both  the  system  and  the  person.  When  there  is  disconnection  between 
organizational  systems  and  people  are  required  to  actively  re-organize  information  into 
appropriate  structures,  they  have  difficulty  preserving  the  relations  among  items  (Durding, 
Becker,  &  Gould,  1977).  As  an  example  of  this  phenomenon  in  submarine  display  development, 
when  there  was  a  transition  from  analog  to  digital  systems,  one  advance  was  to  no  longer  use  the 
line  of  sight  display.  However,  brief  experience  on  board  without  this  cognitively  compatible 
display  resulted  in  negative  feedback  from  operators  and  resulted  in  the  return  of  the  line  of  sight 
display.  This  cognitive  phenomenon  is  related  to  an  activation  model  of  cognition  in  that  when 
primed  with  a  particular  information  item,  people  also  think  of  (or  "activate")  similar 
information.  There  is  a  finite  range  of  activation  around  a  central  concept  and  the  activation 
burden  (i.e.,  non-conscious  retrieval)  is  eased  when  similar  information  is  presented  together 
(Schmitter-Edgecombe,  1999).  Managing  the  level  of  cognitive  activation  is  important  because 
there  is  a  limit  to  human  processing  capacities.  Many  information  processing  theorists  consider 
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too  much  information  a  key  source  of  performance  degradation  (Janis  &  Mann,  1977).  Current 
and  proposed  equipment  makes  it  possible  to  display  any  information  at  CONN,  from  raw 
auditory  data  to  refined  visual  displays  of  ships'  predicted  positions.  In  this  environment  of 
powerful  display  capability,  an  important  consideration  of  the  human  as  part  of  the  system  may 
dictate  that  less  information  be  provided  than  is  technologically  possible  (e.g.,  Human  Systems 
Integration).  An  accurate  cognitive  model  would  serve  as  an  expert  system  to  present  or 
emphasize  the  most  relevant  information. 

The  process  of  identifying  the  most  relevant  information  for  submarining  and  effective 
organization  of  that  information  must  involve  active  duty  submariners,  the  experts.  This 
involvement  of  experts  could  be  through  empirical  assessment  during  real  or  simulated 
operations,  but  such  approaches  are  expensive  and  time-consuming,  especially  when  involving 
many  subjects.  Alternately,  interviews  and  other  consultation  with  submariners  provide  insight 
into  their  cognitive  approach  (Kirschenbaum,  1992;  Soldow  1998),  but  often,  people  do  not  have 
conscious  access  to  their  own  underlying  mental  organization  and  cannot  articulate  rules  for  their 
behavior  (Ashby  &  Maddox,  1992,  1993),  particularly  as  seen  in  a  submarine  navigation,  (Sun, 
Merrill,  &  Peterson,  2001).  Talk-aloud  protocols  (i.e.,  verbalizing  while  going  through  actions) 
can  be  more  revealing  than  direct  interview,  but  they  have  limitations  as  well,  especially  in 
combining  data  across  many  subjects.  Another  research  alternative,  the  one  used  in  this  report,  is 
an  unconstrained  sorting  task  (aka,  Q-sort)  and  nonmetric  multidimensional  scaling  and 
clustering  analyses  (Kruskal,  1964;  Shepard,  1962).  Other  investigators  have  used  these  methods 
exploring  Naval  antisubmarine  warfare  (ASW)  (Laxar,  Moeller,  &  Rogers,  1983;  Laxar,  Rogers, 
&  Moeller,  1988;  Zachary,  1980).  In  this  report,  sorting  and  ranking  tasks  are  used  to  elicit 
judgments  from  Naval  Officers  about  submarine  information,  to  determine  how  such  information 
is  organized  and  assigned  priorities  by  the  submarine  OOD  according  to  level  of  experience. 
These  methods  accommodate  active  duty  submariners'  limited  time  and  afford  aggregation 
across  many  subjects. 

The  operational  community  has  called  for  consideration  of  the  operator  in  systems  design 
(Armbruster,  2003),  and  the  Naval  Sea  Systems  Command  is  addressing  this  concern  by  the 
recently  instantiated  Human  Systems  Integration  directorate  (e.g.,  NAVSEA  03).  The  objective 
of  this  research  is  to  model  the  submarine  OOD's  approach  to  information  they  use  on  board  to 
execute  their  mission.  This  model  can  be  employed  in  systems  design  and  operations  to  serve  as 
an  expert  decision  aid,  and  to  present  the  operator  with  "intuitive"  information  (i.e.,  knowledge). 
An  intuitive  display  can  be  achieved  by  presenting  information  in  an  organization  that  is 
consistent  with  the  operator's  own  cognitive  organization  and  consequent  expectations. 

Without  an  appreciation  of  the  user's  approach  (as  derived  from  the  human-based  cognitive 
model),  the  presentation  of  information  is  left  to  the  discretion  of  the  system  designer  (Kallmeier, 
et  al.,  2001).  A  computer  system  may  handle  vast  amounts  of  information,  but  such  systems 
usually  fall  short  when  determining  what  information  is  important  and  relevant  in  a  particular 
situation,  especially  as  the  situation  changes.  As  a  now  commonplace  example,  World  Wide 
Web  search  engines  can  process  thousands  of  web  pages  in  milliseconds,  but  they  often  present 
an  overwhelming  return  to  the  human  operator,  rather  than  the  specific  item(s)  the  person 
desires.  As  an  alternative  to  burdening  the  operator  by  requiring  additional  complex  inputs  to 


retrieve  relevant  information,  an  understanding  of  the  operator's  approach  can  facilitate  the 
retrieval  of  information  important  for  the  task  at  hand. 


OBJECTIVES 

A  central  theme  of  command  and  control  is  the  power  of  information.  In  discussions  related  to 
"information  superiority",  situational  awareness  is  a  major  element.  Situational  awareness  refers 
to  the  ability  of  an  operator  to  perceive  the  relevant  informational  elements  in  the  environment, 
comprehend  and  integrate  these  various  pieces  of  information  in  support  of  an  operator's  goals, 
and  predict  future  events  and  system  states  based  on  this  understanding  (Endsley,  1995). 
However,  one  of  the  greatest  challenges  facing  operators  and  technology  providers  is  to  match 
our  increasing  capability  to  gather,  process,  and  display  information  useful  for  situation 
awareness  with  a  corresponding  increase  in  our  ability  to  use  that  information.  Information 
overload  is  a  valid  concern  and  may  be  an  unintended  consequence  of  the  quest  for  information 
superiority. 

More  information  does  not  always  mean  better  situational  awareness,  unless  we  gather,  integrate, 
and  present  information  in  a  way  that  supports  the  decision  maker's  cognitive  processes. 
Sometimes  the  addition  of  more  information  may  be  counterproductive  because  it  brings  with  it 
the  potential  for  additional  ambiguities  and  the  need  for  users  to  cognitively  process  greater 
numbers  and  types  of  information  inputs.  Without  the  tools  to  handle  the  increased  size  and 
complexity  of  the  situational  awareness  picture,  operators  will  be  faced  with  the  problem  of 
information  overload.  This  project's  goal  is  to  increase  situational  awareness  in  a  way  that  helps 
interpret  the  increased  knowledge  base  so  that  it  helps  rather  than  hinders  the  decision  making 
process. 

This  research  defines  the  information  required  by  the  submarine  OOD  to  perform  his  duties  and 
specifies  how  this  information  should  be  organized  for  optimal  accessibility  by  the  decision 
maker,  as  in  a  computer-based  retrieval  and  display  system.  These  results  will  serve  as  a  basis  on 
which  to  design  or  refine  a  human  system  interface  for  information  retrieval  and  display  systems 
in  the  submarine  Command  and  Control  Center. 

Specifically,  the  current  research  is  exploratory  and  descriptive  in  nature,  and  addresses  the  issue 
of  how  submariners  cognitively  organize  and  prioritize  information  in  different  operational 
scenarios,  and  whether  these  mental  models  change  as  a  result  of  experience.  These  data  will  be 
compared  with  precursor  research  of  submarine  operators'  cognitive  approach  to  submarine  sonar 
information  by  OODs  (Laxar,  Moeller,  &  Rogers,  1983)  and  by  sonar  technicians  (Laxar, 
Moeller,  &  Rogers,  1989).  The  earlier  research  used  the  same  methodology  as  the  present 
research,  but  the  information  considered  was  limited  to  sonar  information.  In  those 
investigations,  information  was  found  to  be  organized  in  two  dimensions,  sonar  information 
source  (contact  v.  environment)  and  sonar  information  destination  (sonar  v.  CONN).  It  is 
expected  that  the  inclusion  of  non-sonar  information  and  new  technology  (e.g.,  relatively  new 
sensor  capability)  may  reflect  a  different  categorization  of  information  by  OODs. 
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METHODS 


Participants 

Data  were  collected  from  83  Naval  Officers.  Seventy-nine  of  these  participants  were  students  at 
Naval  Submarine  School,  Submarine  Base  New  London,  and  four  were  instructors  at  the  Naval 
War  College,  Newport,  RI.  These  83  participants  comprised  four  groups  in  this  research,  listed 
here  in  ascending  order  of  submarine  expertise.  Submarine  Officer  Basic  Course  (SOBC) 
students  served  as  two  groups  of  'novice'  participants  (N  =  39).  The  SOBC  students  are  not 
OOD-qualified,  but  in  SOBC  they  do  receive  all  the  relevant  lectures  and  simulator  experience  in 
initial  OOD  training.  Data  were  collected  from  the  SOBC  students  during  the  first  week  of  class 
(SOBCpre)  and  the  last  week  of  class  (SOBCpst).  Submarine  Officer  Advanced  Course  (SO AC) 
students  served  as  'experienced'  participants  (N  =  40).  SOAC  students  are  OOD-qualified. 

Finally,  post  Department  Head  submarine  officers  currently  filling  a  shore  tour  billet  served  as 
'expert'  participants  (N  =  4;  Post  DH).  Descriptive  information  for  the  83  participants  is  presented 
in  Tables  1  and  2.  The  career  path  of  submarine  officers  displayed  in  Figure  1  provides  a 
schematic  for  the  level  of  experience  of  the  three  subject  groups  used  in  this  study. 

Table  1.  Subjects' Demographic  Information 


Characteristic 

N 

Mean 

SD 

Age 

SOBC 

39 

25.2 

2.4 

SOAC 

40 

31.2 

2.4 

Post  DH 

4 

40.5 

0.6 

Years  of  Service 

SOBC 

39 

5.1 

3.6 

SOAC 

40 

10.6 

3.5 

Post  DH 

4 

20.4 

1.9 

Years  of  Sea  Duty 

SOBC 

39 

1.1 

1.3 

SOAC 

40 

3.8 

1.2 

Post  DH 

4 

6.5 

0.6 

Table  2.  Rank  Distribution  of  Subjects 

Rank 

01 

02 

03 

04 

05 

Group 

SOBC 

37 

2 

0 

0 

0 

SOAC 

0 

0 

39 

1 

0 

Post  DH 

0 

0 

0 

4 

0 

10 


CART 


MAJOR 

COMMAND 


Figure  1.  Submarine  Officer  career  path.  Participant  groups  used  in  the  study  are  outlined  with  bold  lines. 
Materials 

Various  types  of  information  from  current  and  proposed  submarine  systems  were  classified  into 
20  categories  by  the  investigators  at  the  Naval  Submarine  Medical  Research  Laboratory  and 
submarine  fire  control  experts  at  the  Naval  Undersea  Warfare  Center  (NUWC).  To  build  upon 
the  earlier  research  by  Laxar  et  al.,  (1983,  1989)  which  described  submariners'  cognitive 
organization  of  15  categories  of  sonar  information,  some  of  those  categories  were  included 
within  the  set  of  20  items  with  minimal  change  where  appropriate.  These  20  categories  are  listed 
in  Appendix  A.  Descriptions  of  these  categories  comprised  the  20  stimuli  for  the  tasks  to  be 
performed  by  the  subjects.  Each  of  the  stimuli  was  typed  on  a  separate  3"  X  5"  card  and 
numbered  on  the  reverse  side  to  create  the  stimulus  deck. 

Some  information  available  to  the  OOD  at  CONN  (control  of  a  submarine's  movements;  e.g., 
torpedo  load  out,  weapons  safeties,  countermeasures  status,  intelligence  on  contacts  in  the  area) 
was  not  included  in  this  investigation  because  it  was  more  relevant  to  target  prosecution  than 
situational  awareness.  Several  subjects  did  request  such  information,  which  indicates  its 
importance  for  the  OOD,  but  it  was  considered  outside  the  scope  of  the  present  research. 

Procedure 

All  subjects  were  given  a  brief  overview  of  the  procedure  when  consent  forms  were 
administered.  This  research  was  conducted  in  compliance  with  all  applicable  federal  regulations 
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governing  the  protection  of  human  subjects  in  research.  Next,  participants  completed  a 
background  questionnaire  to  assess  their  level  of  experience  (Appendix  B).  Next,  participants 
judged  the  similarity  between  different  categories  of  information  available  to  the  OOD  at  CONN. 
This  judgment  of  similarity  is  a  reflection  of  the  conceptual  organization  operators  employ.  To 
make  these  similarity  judgments,  subjects  completed  an  unconstrained  sorting  task.  The  officers 
received  a  set  of  20  index  cards  with  a  short  description  of  submarine  information  printed  on 
each  and  then  arranged  these  cards  into  groups  according  to  similarity.  The  definition  of 
similarity  was  left  up  to  the  subject.  Cards  describing  similar  categories  were  placed  in  the  same 
group.  Any  card  describing  a  unique  category  was  placed  by  itself.  The  officers  could  create  as 
many  or  as  few  groups  as  they  felt  appropriate. 

After  the  sorting  task,  the  officers  ranked  the  information  cards  according  to  relative  importance, 
creating  a  single  pile  with  the  most  important  card  on  top  and  the  least  important  card  on  bottom. 
Subjects  ranked  the  items  four  times,  once  for  each  of  four  different  operational  scenarios 
(Appendix  C).  The  factors  that  varied  over  the  four  scenarios  were  the  type  of  contact  (neutral 
vs.  hostile)  and  the  type  of  environment  (deep  water  vs.  shallow  water).  The  first  scenario  was  an 
Antisubmarine  Warfare  (ASW)  patrol  in  a  shallow  water  environment  (i.e.,  littoral  sea;  "brown 
water")  with  a  hostile  contact.  The  second  scenario  was  the  same  as  the  first,  except  the  contact 
was  neutral.  The  third  scenario  was  an  ASW  patrol  in  a  deep  water  environment  (i.e.,  pelagic 
ocean;  "blue  water")  with  a  hostile  contact.  Finally,  the  fourth  scenario  was  the  same  as  the  third 
except  the  contact  was  neutral.  These  four  scenarios  differ  greatly  in  the  quality  and  quantity  of 
information  available  to  the  OOD. 

After  each  task,  subjects  recorded  their  responses  (i.e.,  card  sorting  or  ordering)  on  answer  sheets 
according  to  the  code  number  on  the  back  of  each  stimulus  card. 

RESULTS 

The  card  sorting  data  were  submitted  to  both  multidimensional  scaling  analyses,  to  identify 
underlying  dimensions  or  organizing  principles,  and  a  cluster  analysis,  to  identify  groups  of 
similar  items.  The  revealed  organization  is  indicative  of  submarine  OOD's  approach  to 
information.  The  ranking  data  were  used  to  examine  information  prioritization  within  this 
organization  and  how  this  prioritization  changes  in  conditions  of  contact  type  and  operational 
environment. 

Multidimensional  Scaling  Analysis 

Multidimensional  scaling  (MDS)  was  used  to  reduce  the  amount  of  data  for  easier  interpretation 
and  to  provide  a  visual  representation  of  patterns  of  similarities  among  the  stimuli.  The  input  to 
the  MDS  analysis,  the  raw  data  from  the  unconstrained  sorting  task,  are  nominal  in  nature  and 
are  difficult  to  aggregate  and  analyze  quantitatively.  To  treat  these  data  quantitatively,  each 
subject's  data  are  systematically  transformed  into  a  matrix  of  similarities  among  the  20  items. 
Then,  all  the  subject's  matrices  are  aggregated  into  a  single  matrix  of  similarities.  This  aggregate 
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matrix  is  the  input  for  the  MDS  analyses1.  The  MDS  analysis  constructs  a  configuration  of  the 
stimuli  in  k-dimensional  space  such  that  the  Euclidean  distances  among  the  stimuli  correspond  as 
closely  as  possible  to  the  input  similarities.  This  construction  is  an  iterative  adjustment  process, 
based  on  the  observed  dissimilarity  between  all  pairs  of  stimuli.  The  final  configuration  is  then 
rotated  so  that  the  principal  components  of  the  points  lie  along  coordinate  axes.  The  coordinate 
axes  are  then  used  to  help  determine  the  underlying  psychological  structure  of  the  stimulus 
domain. 

The  appropriate  number  of  dimensions  is  defined  by  the  degree  to  which  additional  dimensions 
account  for  the  similarities  in  the  data.  At  some  number  of  dimensions,  additional  dimensions  in 
the  analysis  do  not  provide  significant  additional  explanatory  power.  This  point  is  determined  as 
the  number  of  dimensions  at  which  the  stress  (i.e.,  goodness  of  fit)  of  the  model  is  acceptable. 
Generally,  stress  greater  than  0.15  is  unacceptable,  meaning  another  dimension  should  be  added; 
stress  less  than  0.10  is  excellent,  meaning  no  further  dimensions  should  be  added  (Kruskal  & 
Wish,  1978).  This  stress  acceptability  benchmark  is  usually  coincident  with  the  "elbow"  in  a 
scree  plot,  a  plot  of  stress  by  dimensions.  Both  quantitative  and  visual  methods  of  stress 
assessment  were  used  here.  The  dimensions  of  similarity  as  revealed  by  the  analysis  are  not 
necessarily  categorical  in  nature.  These  dimensions  can  be  considered  as  continua  with  no 
natural  gaps.  In  the  graphical  output  of  the  MDS  analysis,  the  larger  distances  are  more  accurate. 

The  dimensions,  or  axes,  resulting  from  MDS  are  meaningless  and  the  orientation  of  the  picture 
is  arbitrary.  What  is  important  is  the  relative  position  among  items  and  clusters  of  items.  The 
labels  associated  with  the  dimensions  are  determined  by  the  researchers  according  to  what  makes 
the  best  sense  given  the  subject  material. 

Solution  stress  levels 

In  the  MDS  analyses  of  the  sorting  data,  the  solution  stress  level  for  the  SOBCpre  subjects  was 
0.09  after  19  iterations,  and  the  stress  level  for  the  SOBCpst  subjects  was  0.05  after  22  iterations. 
The  solution  stress  levels  for  the  SOAC  and  PostDH  subjects  were  0.08  after  24  iterations  and 
0.00  after  16  iterations,  respectively.  All  of  these  stress  levels  fall  within  the  acceptable  range. 


Number  of  dimensions 

The  scree  plots  for  all  subject  groups  revealed  two  dimensions  for  each  group's  organization  of 
the  20  submarine  information  items. 


Coordinates 


1  This  procedure  assigns  values  to  the  400  pairs  of  stimuli  according  to  the  number  of  times  subjects  placed  the 
items  in  the  same  groups.  For  these  analyses  (i.e.,  raw  data  transformation,  matrices  aggregation,  multidimensional 
scaling,  clustering),  Anthropac  software  was  used  (Borgatti,  1996a,  1996b). 
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Figures  2  through  5  show  the  MDS  two-dimensional  plots  for  the  SOBCpre,  SOBCpst,  SOAC, 
and  PostDH  subject  groups.  The  coordinates  of  the  information  items  for  all  subject  groups  are 
listed  in  Appendix  D,  while  the  acronyms  used  in  all  the  plots  are  explained  in  Table  3. 

Table  3.  Acronyms  used  in  MDS  plots 


Concept 

Acronym 

Electronic  Support  Measures  contact  data 

ESM 

Own  ship  data 

OS 

Performance  Monitoring/Fault  Location  data 

PMFL 

Ocean  environment  and  navigation  data 

OE 

Counterdetectability  data 

CD 

Propulsion  Plant  Lineup 

PROP 

Sonar  search  plan 

SSRCHPLN 

Visual  contact  data 

VIS 

Potable  water  status 

POT 

Active  sonar  lineup 

ACT 

Ship's  atmosphere 

ATM 

Sonar  tracker/cursor  audio 

SAUD 

Sanitary  tank  status 

SAN 

Geosit/ops  summary 

GEO 

Trial  own  ship 

TOS 

Non-target  fire  control  solution 

NTFCS 

Sonar  detection  displays 

SDIS 

Sonar  lineup 

SLINUP 

Target  fire  control  solution 

TCS 

Sonar  class  displays 

SCLASDIS 
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Figure  2.  The  two-dimensional  solution  for  the  multidimensional  scaling  analysis  for  information 
available  to  the  submarine  OOD,  SOBCpre  subjects. 
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Figure  3.  The  two-dimensional  solution  for  the  multidimensional  scaling  analysis  for  information 
available  to  the  submarine  OOD,  SOBCpst  subjects. 
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Figure  4.  The  two-dimensional  solution  for  the  multidimensional  scaling  analysis  for  information 
available  to  the  submarine  OOD,  SOAC  subjects. 
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Figure  5.  The  two-dimensional  solution  for  the  multidimensional  scaling  analysis  for  information 
available  to  the  submarine  OOD,  Post  DH. 

Interpretation  of  the  MDS  Results 

For  the  four  subject  groups,  the  horizontal  dimension  in  the  figures  is  interpreted  to  be  the 
destination  (or  primary  user)  of  the  information:  CONN  v.  sonar  (e.g.,  Geo  plot  v.  sonar  lineup). 
Trial  own  ship  (left  side  of  figures)  is  an  aggregation  of  sonar  information  taken  from  successive 
maneuvers  by  own  ship.  The  sonar  gram  (right  side  of  figures)  is  the  visual  representation  of  the 
acoustic  environment  and  is  the  primary  source  of  raw  data  for  the  submarine.  These  types  of 
information  are  delimiters  for  the  ends  of  the  horizontal  axis  in  the  graph  above.  Both  CONN 
and  sonar  benefit  from  knowing  the  environmental  conditions  -  that  is  why  it  falls  halfway 
between.  This  dimension  is  consistent  with  Laxar  et  al.  (1983;  1989).  Those  earlier  findings, 
however,  included  only  sonar  information,  so  there  is  an  additional  quality  to  the  interpretation 
of  this  dimension  in  that  it  also  captures  single  source  information  (or  raw  data;  e.g.,  sonar)  v. 
fused  information  from  multiple  sources  (e.g.,  Geo  plot).  The  information  fusion  quality  of  the 
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interpretation  was  not  as  apparent  in  the  previous  investigation  that  only  dealt  with  sonar 
information,  but  it  is  consistent  with  the  type  of  information  that  would  be  needed  at  CONN. 

The  vertical  dimension  in  the  figures  is  interpreted  to  be  the  source  of  information:  context/noise 
v.  contact/signal  (e.g.,  information  about  the  ocean  environment  v.  intercept  of  active  sonar 
pulse).  The  ocean  environment  and  the  operation  of  the  sonar  system  (i.e.,  PMFL)  are  noise  that 
must  be  factored  out  in  sensor  operation  to  enhance  signal  detection.  Contact  classification  from 
an  intercepted  active  sonar  signal  from  the  contact  and  visual  image  of  the  contact  from  the 
photonics  mast  are  clear  signals  from  the  contact  of  interest.  These  types  of  information  are 
delimiters  for  the  ends  of  the  vertical  axis  in  the  MDS  analysis.  The  Geo  plot  (i.e.,  fusion 
display)  contains  both  noise  and  signal,  and  thus  should  fall  between  the  ends  of  the  horizontal 
axis.  As  with  the  first  dimension,  this  dimension  is  consistent  with  findings  in  Laxar  et  al.  (1983, 
1989). 

Sorting  data  of  all  four  subject  groups  revealed  two  dimensions  for  the  organization  of  the  20 
submarine  information  items.  The  number  of  dimensions  with  which  subjects  mentally  organized 
the  information  items  did  not  differ  between  levels  of  experience.  The  initial  cognitive 
organization  that  SOBC  students  use  to  classify  the  information  does  not  appear  to  be 
significantly  different  from  the  more  experienced  subjects  in  the  SOAC  and  Post  DH  groups, 
with  respect  to  general  organizing  principles. 

The  primary  task  onboard  submarines  is  localization  of  sensed  contacts  (Kirschenbaum,  1992). 

In  this  alien  environment  in  which  visual  creatures  are  deprived  of  visual  information  from  a 
natural  scene,  the  OOD  cannot  simply  scan  the  horizon  to  get  an  understanding  of  where  objects 
are  in  the  environment.  Instead,  data  are  collected  from  the  surrounding  external  environment  by 
different  sensors  (e.g.,  sonar)  and  the  OOD  builds  a  mental  representation  of  the  submarine's 
situation.  This  mental  construction  can  be  difficult  in  an  everyday  environment  (Carr  &  Roskos- 
Ewoldsen,  1999);  it  is  extraordinarily  difficult  in  the  submarine  environment  in  which  everything 
must  be  imagined  (as  opposed  to  being  seen)  and  most  everything  is  in  motion.  Due  to  the 
simultaneous  use  of  different  sensors,  after  initial  data  collection,  the  subsequent  task  is 
resolution  of  the  contacts  that  have  been  localized  by  different  sensors.  This  resolution  helps  the 
OOD's  mental  model  (Johnson-Laird,  1993)  and  situational  awareness  (i.e.,  localization).  This 
situational  awareness  or  mental  model  allows  the  OOD  to  make  decisions  regarding  the 
operation  of  the  ship.  The  two  dimensions  revealed  in  these  data  make  sense  given  the  submarine 
problem.  One  dimension  corresponds  to  sensing  and  localizing  the  contacts  in  the  environment. 
The  other  dimension  corresponds  to  integrating  information  from  different  sources  -  for  the 
same  purpose  as  the  OOD's  mental  model,  to  achieve  situational  awareness  and  to  facilitate 
decision  making  regarding  the  operation  of  the  ship  (i.e.,  targeting,  tracking,  maneuvering). 

Cluster  Analysis 

MDS  analyses  reveal  general  organizing  principles  at  a  gross  level  of  detail.  The  grouping  of 
items  within  organizing  dimensions  is  a  finer-grained  approach  to  understanding  cognitive 
organization  and  may  reveal  a  more  subtle  effect  of  experience.  For  this  finer-grained 
examination  of  the  data,  a  cluster  analysis  was  conducted  to  reveal  natural  groupings  in  the  data. 
Johnson's  (1967)  average  method  which  begins  with  many  small  clusters  and  gradually  merges 
them  into  bigger,  fewer  clusters  was  used.  The  clusters  are  determined  by  Euclidean  distance 
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between  items  and  each  cluster  iteration  groups  the  two  items  that  are  most  proximal.  When  the 
distance  determination  involves  multiple  items  that  have  already  been  grouped  into  a  cluster,  the 
distance  between  a  cluster  and  an  item  or  another  cluster  can  be  determined  by  using  the  smallest 
distance,  the  largest  distance,  or  the  average  or  median  of  distances.  In  this  analysis,  a 
computation  based  on  the  average  of  the  distances  between  clusters,  as  opposed  to  the  minimum 
or  maximum  distance,  was  used.  The  data  input  into  the  analysis  are  the  card  sorting  data 
produced  by  the  subjects  -  the  same  input  as  used  in  the  MDS  analysis.  The  results  of  the  cluster 
analysis  only  provide  the  number  of  clusters  needed  to  best  capture  the  data,  not  the  labels  of  the 
clusters.  The  researchers  give  the  categories  labels  post  hoc  to  best  explain  the  data. 

Cluster  analysis  results  for  SOBCpre  and  SOBCpst  subjects 

For  all  subject  groups,  the  level  of  .50  was  used  as  the  criterion  in  determining  clusters.  This 
means  that  the  average  distance  among  items  within  the  clusters  is  .50  or  less  (maximum  is 
1.00).  Tables  3  and  4  list  the  categories  that  emerged  from  the  cluster  analysis  for  the  SOBCpre 
and  SOBCpst  subjects.  This  group  of  subjects  used  five  groups  to  organize  the  information,  and 
we  labeled  the  categories  as  indicated  in  Tables  3  and  4.  Even  though  the  number  of  clusters 
needed  to  organize  the  information  did  not  change  as  a  result  of  the  10  week  SOBC  course,  the 
placement  of  three  of  the  information  items  did  change.  The  "trial  own  ship"  item  moved  from 
the  Own  Ship  Parameters  cluster  to  the  Fire  Control  Data  cluster,  the  "geosit/ops  summary"  item 
moved  from  the  Contact  Management  cluster  to  the  Fire  Control  Data  cluster,  and  finally,  the 
"ocean  environment  and  navigation  data"  item  moved  from  the  Contact  Management  cluster  to 
the  Sonar  cluster.  It  appears  that  the  Fire  Control  Data  cluster  gained  two  new  items  and  lost  one 
item,  suggesting  some  cognitive  reorganization  of  the  information  items  at  the  local  (vice  global) 
level. 
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Table  3.  Cluster  analysis  results  for  SOBCpre  subjects. 


_ Category _ Item _ 

Own  Ship  Parameters  Own  ship  data 
Trial  own  ship 
Counterdetectability  data 
Ship  System  Status  PMFL  data 

Propulsion  Plant  Lineup 
Ship's  atmosphere 
Potable  water  status 
Sanitary  tank  status 

Contact  Management  ESM  contact  data 
Visual  contact  data 
Geosit/ops  summary 
Ocean  environment  and  navigation  data 
Fire  Control  Data/Target  Analysis  Non-target  fire  control  solution 

Target  fire  control  solution 
Sonar-  Sonar-  search  plan 
Active  sonar-  lineup 
Sonar-  lineup 

Sonar-  tracker/cursor  audio 


Sonar-  detection  displays 
Sonar-  class  displays 


Table  4.  Cluster  analysis  results  for  SOBCpst  subjects. 


Category _ Item _ 

Own  Ship  Parameters  Own  ship  data 

Counterdetectability  data 
Ship  System  Status  PMFL  data 

Propulsion  Plant  Lineup 
Ship's  atmosphere 
Potable  water  status 
Sanitary  tank  status 

Contact  Management  ESM  contact  data 
Visual  contact  data 

Fire  Control  Data  Geosit/ops  summary 
Trial  own  ship 

Non-target  fire  control  solution 
Target  fire  control  solution 
Sonar  Ocean  environment  and  navigation  data 
Sonar-  search  plan 
Active  sonar-  lineup 
Sonar  lineup 

Sonar-  tracker/cursor  audio 


Sonar  detection  displays 
Sonar  class  displays 
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Cluster  analysis  results  for  SOAC  subjects 


The  cluster  analysis  results  for  the  SOAC  subjects  are  displayed  in  Table  5.  An  interesting 
change  occurred  between  the  SOBC  subjects  and  the  SOAC  subjects,  suggesting  that  experience 
plays  an  important  role  in  this  cognitive  organization.  Even  though  SOAC  subjects  are  similar  to 
SOBC  subjects  in  that  they  organized  the  information  items  along  two  dimensions  (as  revealed 
by  the  MDS  results)  and  categorized  the  items  into  five  groups,  the  specific  groupings  changed. 
The  "ocean  environment  and  navigation  data"  item  changed  groups  again,  but  more  interesting, 
the  SOAC  subjects  broke  apart  one  group  into  two  groups,  and  merged  two  other  groups  into  one 
group.  The  Sonar  group  splintered  into  the  Equipment  Lineup/Search  Data  group  and  the  Sonar 
Tactical  Displays  group,  indicating  some  reorganization  of  the  sonar  information.  Additionally, 
the  Contact  Management  group  and  the  Fire  Control  Data  group  merged  into  one  group  for  the 
SOAC  subjects.  In  one  case,  experience  has  taught  the  subjects  that  items  thought  to  be 
dissimilar  are  in  fact  similar,  and  other  items  that  were  thought  to  be  similar  require  a  different 
classification.  This  change  suggests  a  reorganization  of  the  mental  model  they  use  to  interpret  the 
data  as  they  encounter  while  acting  as  OOD. 

Table  5.  Cluster  analysis  results  for  SOAC  subjects. 


Category 

Item 

Own  Ship  Parameters 

Own  ship  data 

Ocean  environment  and  navigation  data 
Counterdetectability  data 

Ship  System  Status 

PMFL  data 

Propulsion  Plant  Lineup 

Ship's  atmosphere 

Potable  water  status 

Sanitary  tank  status 

Contact  Management 

ESM  contact  data 

Visual  contact  data 

Geosit/ops  summary 

Trial  own  ship 

Non-target  fire  control  solution 

Target  fire  control  solution 

Equipment  Lineup/Search  Data 

Sonar  search  plan 

Active  sonar  lineup 

Sonar  lineup 

Sonar  Tactical  Displays 

Sonar  tracker/cursor  audio 

Sonar  detection  displays 

Sonar  class  displays 

Cluster  analysis  results  for  Post  Department  Head  subjects 

Table  6  provides  the  cluster  analysis  results  for  the  most  experienced  subjects,  the  Post 
Department  Head  subjects.  A  new  cognitive  organization  emerges  from  this  analysis,  even 
though  these  subjects  also  required  two  dimensions  to  globally  organize  the  information 
(revealed  by  the  MDS  results).  First,  they  only  used  four  groups  to  organize  the  20  information 
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items,  whereas  the  other  three  subject  groups  used  five  groups.  Second,  items  changed  groups 
from  the  SOAC  classification  to  the  Post  DH  classification.  Specifically,  the  Own  Ship 
Parameters  group  and  the  Contact  Management  group  merged  into  one  category  for  the  Post  DH 
subjects  (even  though  two  items  moved  to  the  Equipment  Lineup/Search  Data  group).  The 
"PMFL"  item  moved  from  the  Ship  System  Status  group  to  the  Sonar  Tactical  Displays  group. 

Table  6.  Cluster  analysis  results  for  Post  DH  subjects. 


Item 


Category 


Own  Ship  Parameters/Contact  Management 


Ship  System  Status 


Equipment  Lineup/Search  Data 


Sonar  Tactical  Displays 


Own  ship  data 
ESM  contact  data 
Visual  contact  data 
Geosit/ops  summary 
Trial  own  ship 

Non-target  fire  control  solution 
Target  fire  control  solution 
Propulsion  Plant  Lineup 
Ship's  atmosphere 
Potable  water  status 
Sanitary  tank  status 
Counterdetectability  data 
Sonar  search  plan 
Active  sonar  lineup 
Sonar  lineup 

Ocean  environment  and  navigation  data 
PMFL  data 

Sonar  tracker/cursor  audio 

Sonar  detection  displays 

Sonar  class  displays _ 


Summary  of  cluster  analysis  results 

The  results  from  the  cluster  analyses  garner  support  from  the  literature  on  expertise.  Several 
studies  have  found  that  as  experience  increases,  the  underlying  cognitive  organization  changes  in 
a  way  that  increases  semantic  connections  between  subject  matter  content  (Fiore,  Fowlkes, 
Martin-Milham,  &  Oser,  2000).  In  the  current  study,  it  was  found  that  the  most  experienced 
participants  (PostDH)  utilized  four  clusters  to  categorize  the  information  items,  while  the  other 
groups  required  five  clusters.  Even  though  the  other  three  subject  groups  used  five  clusters,  the 
specific  placement  of  the  items  within  the  clusters  varied  as  a  function  of  experience.  Because 
some  of  these  items  (e.g.,  ocean  environment  and  navigation  data,  PMFF,  etc.)  changed  groups 
across  different  levels  of  experience,  we  would  expect  that  these  items  would  also  move  along 
the  two  dimensions  of  context/noise  v.  contact/signal  and  CONN  v.  sonar  -  the  destination  (or 
primary  user)  of  the  information. 

The  MDS  and  cluster  analyses  provide  insight  into  how  submarine  Officers  cognitively  organize 
information  in  a  mental  model  in  a  generic  context.  The  priority  ranking  task,  on  the  other  hand, 
provides  us  with  situation-specific  mental  models. 
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Ranking  Tasks 

Participants  ranked  the  20  information  items  according  to  priority  for  four  operational  scenarios. 
For  the  ranking  task  data,  the  data  could  be  presented  in  many  different  ways.  Given  the 
abundance  of  the  data  (4  scenarios  for  4  groups  =  16  lists),  the  data  were  consolidated  into  some 
meaningful  comparisons.  Ranking  data  for  all  four  subject  groups  across  the  four  scenarios  are 
presented  in  Appendix  E. 

Effect  of  experience  on  littoral  and  pelagic  environment  rankings 

The  scenarios  varied  across  operating  environment,  littoral  vs.  pelagic,  and  type  of  contact, 
hostile  vs.  neutral.  First,  a  closer  look  is  taken  at  the  effect  of  experience  on  prioritization  of  the 
information  items  across  environments  and  contact  situations.  The  change  in  judged  importance 
among  the  20  submarine  information  items  for  the  pelagic  and  littoral  operating  environments 
across  Post  DH,  SOAC,  and  SOBC  post  participants  is  shown  in  Figures  6  and  7.  In  this  analysis 
SOBCpre  subjects  were  not  included  because  it  was  determined  that  the  most  value  would  be 
gained  from  comparisons  among  the  other  groups. 

Pelagic  Environment 


Visual  Contact  Data 
ESM  Contact  Data 
TFCS 
Trial  Own  Ship 
Sanitary  Tank  Status 
Sonar  Detection  Displays 
Own  Ship  Data 
Geosit/Ops  Summary 
Sonar  Search  Plan 
NTFCS 
Ocean  Env/Nav  Data 
Sonar  Tracker/Cursor  Audio 
Active  Sonar  Lineup 
Ship's  Atmosphere 
Potable  Water  Status 
Propulsion  Plant  Lineup 
Sonar  Lineup 
PMFL  Data 
Sonar  Class  Displays 
Counterdetectability  Data 
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Figure  6.  Change  in  judged  importance  among  the  20  information  items  between  PostDH,  SOAC,  and 
SOBCpst  participants  for  the  pelagic  environment.  Note:  Negative  numbers  mean  that  PostDH  subjects 
rated  that  item  as  more  important,  while  positive  numbers  mean  that  PostDH  subjects  rated  the  item  as 
less  important. 
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For  the  pelagic,  or  deep  water,  operating  environment,  major  differences  were  found  in 
prioritization  for  several  of  the  information  items.  The  most  experienced  subjects  judged  several 
items  as  more  important  than  the  other  subject  groups.  These  include  counterdetectability  data, 
sonar  class  displays,  PMFL  data,  and  sonar  lineup.  They  also  judged  several  items  as  less 
important  than  other  subject  groups,  including  visual  contact  data,  ESM  contact  data,  TFCS,  and 
trial  own  ship. 


Littoral  Environment 


Visual  Contact  Data 
ESM  Contact  Data 
TFCS 

Propulsion  Plant  Lineup 
Sanitary  Tank  Status 
Own  Ship  Data 
Trial  Own  Ship 
Geosit/Ops  Summary 
Sonar  Search  Plan 
Active  Sonar  Lineup 
Sonar  Detection  Displays 
PMFL  Data 
NTFCS 

Counterdetectability  Data 
Potable  Water  Status 
Ship's  Atmosphere 
Ocean  Env/Nav  Data 
Sonar  Class  Displays 
Sonar  Tracker/Cursor  Audio 
Sonar  Lineup 
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■  Post  DH  to  SOBC 
□  Post  DH  to  SOAC 


Figure  7.  Change  in  judged  importance  among  the  20  information  items  between  PostDH,  SOAC,  and 
SOBCpost  participants  for  the  littoral  environment.  Note:  Negative  numbers  mean  that  PostDH  subjects 
rated  that  item  as  more  important,  while  positive  numbers  mean  that  PostDH  subjects  rated  the  item  as 
less  important. 

For  the  littoral,  or  shallow  water,  operating  environment,  major  differences  were  found  in 
prioritization  for  several  of  the  information  items.  Once  again,  the  Post  Department  Head 
subjects  deemed  sonar  lineup,  sonar  tracker/cursor  audio,  sonar  class  displays,  and  ocean 
environment  more  important  than  the  other  groups.  Visual  contact  data,  ESM  contact  data,  and 
TFCS  items  were  rated  as  being  less  important  for  the  littoral  than  the  less  experienced  subjects 
ranked  them. 
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Effect  of  experience  on  neutral  and  hostile  contact  rankings 


The  change  in  judged  importance  among  the  20  submarine  information  items  for  the  neutral  and 
hostile  contact  scenarios  across  Post  DH,  SOAC,  and  SOBC  post  participants  is  shown  in 
Figures  8  and  9. 


Neutral  Contact 


ESM  Contact  Data 
Visual  Contact  Data 
TFCS 
Trial  Own  Ship 
NTFCS 
Active  Sonar  Lineup 
Sanitary  Tank  Status 
Own  Ship  Data 
Sonar  Search  Plan 
Geosit/Ops  Summary 
Sonar  Detection  Displays 
Ocean  Env/Nav  Data 
Ship's  Atmosphere 
PMFL  Data 
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Counterdetectability  Data 
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Figure  8.  Change  in  judged  importance  among  the  20  information  items  between  PostDH,  SOAC,  and 
SOBCpost  participants  for  the  neutral  contact  scenario.  Note:  Negative  numbers  mean  that  PostDH 
subjects  rated  that  item  as  more  important,  while  positive  numbers  mean  that  PostDH  subjects  rated  the 
item  as  less  important. 

For  the  situations  in  which  there  is  a  neutral  contact.  Post  DH  subjects  prioritized  the  information 
items  differently  than  the  other  subject  groups.  Sonar  tracker/cursor  audio,  propulsion  plant 
lineup,  counterdetectability  data,  and  sonar  class  displays  were  given  more  emphasis  for  the  Post 
DH  subjects.  In  contrast,  ESM  data,  visual  contact  data,  TFCS,  and  trial  own  ship  were  given 
higher  priority  by  the  less  experienced  subjects. 
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Hostile  Contact 


Visual  Contact  Data 
ESM  Contact  Data 
Own  Ship  Data 
Trial  Own  Ship 
Ocean  Env/Nav  Data 
Sanitary  Tank  Status 
Sonar  Detection  Displays 
TFCS 
NTFCS 

Counterdetectability  Data 
Sonar  Search  Plan 
Propulsion  Plant  Lineup 
Sonar  Tracker/Cursor  Audio 
Ship's  Atmosphere 
Potable  Water  Status 
Geosit/Ops  Summary 
Active  Sonar  Lineup 
Sonar  Class  Displays 
PMFL  Data 
Sonar  Lineup 
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Figure  9.  Change  in  judged  importance  among  the  20  information  items  between  PostDH,  SOAC,  and 
SOBCpost  participants  for  the  hostile  contact  scenario.  Note:  Negative  numbers  mean  that  PostDH 
subjects  rated  that  item  as  more  important,  while  positive  numbers  mean  that  PostDH  subjects  rated  the 
item  as  less  important. 

Figure  9  reveals  that  for  the  situations  in  which  a  hostile  contact  is  present,  the  prioritization  of 
items  changes  with  experience,  similar  to  the  differences  in  the  neutral  contact  situation.  Post  DH 
subjects  ranked  sonar  lineup,  PMFL  data,  and  sonar  class  displays  as  more  important,  while 
visual  contact  data,  ESM  contact  data,  and  own  ship  data  were  ranked  more  important  by  the  less 
experienced  groups. 

While  the  data  were  analyzed  according  to  the  two  operating  environments  and  two  contact 
situations,  a  pattern  emerged  in  comparing  the  most  experienced  submariners  with  the  less 
experienced  submariners.  Post  DH  participants  placed  a  greater  emphasis  on  PMFL  data,  sonar 
lineup,  sonar  class  displays,  and  counterdetectability  data  items  in  each  situation.  Compared  to 
the  Post  DH  subjects,  the  SOAC  and  SOBC  participants  consistently  prioritized  visual  contact 
data,  ESM  contact  data,  TFCS,  and  trial  own  ship  as  more  important. 

Changes  between  littoral  and  pelagic  environment  rankings 

Next,  a  closer  look  is  taken  at  the  effect  of  operating  environment  and  contact  situation  for  the 
more  experienced  participants,  SOAC  and  Post  DH  subjects.  These  subject  groups  were  used 
because  they  would  provide  insight  into  an  "expert"  mental  model  that  the  SOBC  subjects  have 
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not  yet  acquired.  However,  the  two  groups  were  not  combined  given  the  differences  in  the  way 
they  prioritized  the  data  in  the  preceding  analysis. 

Figure  10  displays  the  changes  in  prioritization  of  the  information  items  between  the  littoral  and 
pelagic  operating  environments.  The  figure  breaks  down  this  difference  for  both  Post  DH  and 
SOAC  subjects. 


Littoral  vs.  Pelagic  Environments 


ESM  Contact  Data 
Sonar  Tracker/Cursor  Audio 
Visual  Contact  Data 
Ocean  Env/Nav  Data 
Ship's  Atmosphere 
Trial  Own  Ship 
TFCS 

Potable  Water  Status 
Sonar  Search  Plan 
Sonar  Lineup 
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Own  Ship  Data 
Sonar  Detection  Displays 
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Figure  10.  Change  in  judged  importance  among  the  20  information  items  between  littoral  and  pelagic 
operating  environments  for  PostDH  and  SOAC  participants.  Note:  Positive  numbers  mean  that  the  item 
was  rated  as  more  important,  while  negative  numbers  mean  that  the  item  was  rated  as  less  important  in 
the  pelagic  environment. 

For  the  SOAC  subjects,  there  was  no  noticeable  difference  in  the  prioritization  of  the  items  for 
the  pelagic  and  littoral  operating  environments.  However,  for  the  Post  DH  subjects,  major 
differences  did  emerge.  For  the  pelagic  environment.  Post  DH  subjects  rated  propulsion  plant 
lineup,  counterdetectability  data,  and  PMFL  data  as  more  important  than  for  the  littoral 
environment.  Items  deemed  less  important  include  ESM  contact  data,  sonar  tracker/cursor  audio 
and  visual  contact  data.  These  results  suggest  that  serving  as  a  department  head  onboard  a 
submarine  provides  a  unique  experience  that  alters  the  manner  in  which  the  OOD  prioritizes  data 
for  different  situations. 
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Changes  between  hostile  and  neutral  contact  rankings 


Figure  1 1  shows  the  changes  in  prioritization  of  the  information  items  between  the  hostile  and 
neutral  contact  situation.  As  before,  the  data  are  broken  down  for  Post  DH  and  SO  AC  subjects. 


Hostile  vs.  Neutral  Contact 


Sonar  Class  Displays 
Sonar  Detection  Displays 
Counterdetectability  Data 
Geosit/Ops  Summary 
Sonar  Tracker/Cursor  Audio 
Propulsion  Plant  Lineup 
Sonar  Lineup 
Own  Ship  Data 
Ocean  Env/Nav  Data 
NTFCS 
TFCS 

Potable  Water  Status 
Sonar  Search  Plan 
PMFL  Data 
Ship's  Atmosphere 
Trial  Own  Ship 
Active  Sonar  Lineup 
Sanitary  Tank  Status 
ESM  Contact  Data 
Visual  Contact  Data 
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Figure  11.  Change  in  judged  importance  among  the  20  information  items  between  hostile  and  neutral 
contact  operating  environments  for  PostDH  and  SOAC  participants.  Note:  Positive  numbers  mean  that 
the  item  was  rated  as  more  important  in  the  neutral  contact  environment,  while  negative  numbers  mean 
that  the  item  was  rated  as  less  important. 

There  were  minor  differences  in  prioritization  of  the  data  for  the  SOAC  subjects  when 
comparing  the  hostile  and  neutral  contact  situations.  However,  extreme  differences  were  found 
for  the  Post  DH  subjects.  Visual  contact  data,  ESM  contact  data,  sanitary  tank  status,  and  active 
sonar  lineup  were  judged  as  more  important  in  the  neutral  contact  situation,  while  sonar  class 
displays,  sonar  detection  displays,  and  counterdetectability  data  were  judged  as  more  important 
in  the  hostile  contact  situation. 

In  both  comparisons,  the  different  operating  environments  produced  a  much  larger  difference  for 
the  Post  DH  subjects  than  the  SOAC  subjects.  This  suggests  that  the  increase  in  experience  from 
the  SOAC  level  to  the  Post  DH  level  provides  an  opportunity  to  master  the  domain,  which  results 
in  differences  in  how  the  subject  matter  is  organized  and  prioritized. 
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DISCUSSION 


This  study  was  an  investigation  into  the  cognitive  organization  of  submarine  OOD  mental 
models,  and  how  they  may  change  with  experience  and  with  the  operational  scenario.  The 
multidimensional  scaling  analysis  revealed  that  the  20  submarine  information  items  are 
organized  along  two  dimensions,  source  of  the  information  (signal  v.  noise)  and  user  of  the 
information  (sonar  v.  CONN).  This  two-dimensional  organization  did  not  change  as  a  result  of 
experience;  however,  the  manner  into  which  the  information  items  were  categorized  on  a  finer- 
grained  level  did  change  with  experience.  The  most  experienced  participants,  the  Post  DH 
submariners,  used  four  categories  to  organize  the  20  items,  while  the  less  experienced  subject 
groups  used  five  categories.  Over  the  course  of  experience,  the  placement  of  some  of  the 
information  items  changed,  some  groups  merged,  and  some  groups  split  into  two  groups  (e.g., 
the  sonar  information). 

The  prioritization  task  for  the  four  operational  scenarios  provided  a  finer  analysis  of  the  mental 
models  the  submarine  OOD  possesses  as  a  function  of  experience,  and  as  a  function  of  scenario. 
Interestingly,  the  difference  in  prioritization  between  the  SOBC  (novice)  and  SOAC 
(intermediate)  student  participants  was  not  as  great  as  expected.  The  difference  between  these 
two  groups  and  the  Post  DH  participants  was  very  noticeable,  however.  There  appears  to  be  a 
change  in  the  mental  models  of  submarine  officers  between  the  time  they  are  SOAC  students, 
and  several  years  later  after  they  have  served  a  department  head  tour.  The  Post  DH  participants 
had  about  twice  as  many  years  in  the  Navy,  and  twice  as  much  time  on  sea  duty,  which  may 
explain  the  big  difference. 

This  analysis  investigated  group  differences,  and  does  not  address  the  issue  of  individual 
differences  within  the  subject  groups.  A  predominant  theory  of  expertise  suggests  that  as  people 
become  more  experienced  in  a  particular  domain,  their  mental  models  become  more 
homogenous  (Fiore,  Fowlkes,  Martin-Milham,  &  Oser,  2000).  In  other  words,  how  much  do  the 
individuals  within  the  group  agree  with  the  aggregate  group  model?  This  issue  is  addressed  in 
Shobe,  Fiore,  &  Carr  (2004). 


CONCLUSIONS 

There  are  several  ways  in  which  these  results  could  be  used  for  the  operational  community.  First, 
the  design  of  decision  aids  and  other  electronic  combat  systems  onboard  the  submarine  could  be 
aided  with  this  knowledge  of  how  submariners  would  intuitively  think  about  and  organize  the 
information  that  is  being  conveyed  to  them.  Second,  concerning  the  effect  of  experience  on  these 
mental  models,  training  commands  could  make  use  of  the  expert  model  and  teach  toward  it  for 
the  junior  officers.  One  limitation  of  this  study  is  that  we  did  not  correlate  the  mental  models 
with  indices  of  performance.  There  is  a  strong  possibility  that  submariners  better  at  their  job 
would  possess  a  different  cognitive  organization  than  average  or  poor  submariners. 
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APPENDIX  A  Information  Categories  Available  to  the  OOD  at  CONN 


1.  ESM  contact  data 

Bearing  and  classification  of  radar  and  communications  systems  data 
provided  by  the  Electronic  Support  Measures  system 

2.  Own  ship  data 

Own  ship's  course,  speed,  depth,  etc. 

3.  PMFL  data 

Data  on  fault  location  and  performance  monitoring  of  various  sonar 
components,  i.e.,  data  required  to  maintain  electrical  information  systems 

4.  Ocean  environment 
and  navigation  data 

Sea  state;  bottom  depth;  type,  and  features;  background  noise;  sound  velocity 
profile;  propagation  loss,  i.e.,  computer  generated  data  used  to  position  own 
ship  to  maximize  sonar  detection  range  (Le) 

5.  Counterdetectability 
data 

Own  ship  electromagnetic  signature,  radiated  noise,  surface  visibility/wake, 
contaminants,  etc. 

6.  Propulsion  Plant 

Lineup 

Operating  condition  of  the  reactor  and  propulsion  equipment  including  reactor 
coolant  pump  speed  (fast,  slow,  RFO);  electric  plant  lineup  (full  power,  half 
power,  other);  any  limits  on  operations  due  to  casualties  or  maintenance  in  the 
engineering  spaces 

7.  Sonar  search  plan 

Physical  lineup  of  sonar  system,  including  operator  displays  and  processing 
used;  search  track  to  be  followed  by  ship  to  locate  target  of  interest;  depth  to 
search  at  based  on  ocean  bathymetric  data 

8.  Visual  contact  data 
(from  photonics  mast) 

Visual  information  on  surface  contacts  including  night  vision,  infrared,  video, 
laser  ranging  etc.  (Note:  only  available  at  PD) 

9.  Potable  water  status 

Amount  of  water  in  potable  water  tanks  and  tank  on  service 

10.  Active  sonar  lineup 

Power,  pulse  type,  range  gate,  etc.,  i.e.,  parameters  indicating  active  sonar 
transmission  status 

11.  Ship's  atmosphere 

Status  of  atmosphere  inside  the  submarine  (oxygen  percentage,  C02 
concentration  percentage,  CO/PE  concentrations);  status  of  oxygen  bleed  from 
02  banks;  status  of  oxygen  generator  or  02  candle  furnace;  status  of  C02 
scrubbers  and  CO/H2  burner 

12.  Sonar  tracker/cursor 
audio 

Auditory  presentation  of  minimally  processed  sonar  signals  as  picked  up  by 
the  various  arrays 

13.  Sanitary  tank  status 

Tank  levels  in  ship's  sanitary  tanks  and  status  of  pumping  or  blowing  tanks 

14.  Geosit/ops  summary 

Computer  generated  geographical  picture  of  all  contacts  and  own  ship,  in 
either  true  or  relative  bearing  orientation,  with  classification  information 
where  possible 

15.  Trial  own  ship 

CPA  solutions,  trial  maneuvers,  etc.,  i.e.,  data  which  aid  in  assessment  of  the 
present  and  future  tactical  situation 

16.  Non-target  fire 
control  solution 

Rapid  passive  localization  by  wide  aperture  array  (WAA),  KAST  ranging, 
multipath  ranging,  D/E  ranging,  hyperbolic  ranging;  Bearing,  range,  course, 
speed,  depth  if  submerged  contact,  for  secondary  contacts 

17.  Sonar  detection 
displays 

Visual  displays  of  sonar  detections  as  presented  to  sonar  operators 

18.  Sonar  lineup 

Narrow  band  and  broadband  detection  system  lineup,  filter  and  alarm  settings, 
other  operator  set  parameters 

19.  Target  fire  control 
solution 

Rapid  passive  localization  by  wide  aperture  array  (WAA),  KAST  ranging, 
multipath  ranging,  D/E  ranging,  hyperbolic  ranging;  Bearing,  range,  course, 
speed,  depth  if  submerged  contact,  for  target  of  interest 

20.  Sonar  class  displays 

Sonar  signal  interpretation  aids  such  as  signature  assemblies,  lofargrams,  etc. 
i.e.,  data  used  to  aid  in  the  classification  of  signals 
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APPENDIX  B  Background  Survey 

Please  provide  responses  to  the  following  to  help  us  evaluate  the  results  of  this  experiment. 

1 .  Present  rank _ 

2.  Years  in  rank _ 

3.  Age _ 

4.  Years  in  service  _ 

5.  Years  Sea  Duty  _ 

6.  Sea  Billets _ 


7.  Present  billet _ _ 

8.  Training  (e.g.,  SOBC,  SOAC,  JO  courses,  PXO,  PCO) 


36 


APPENDIX  C  Scenarios  Used  for  Prioritization  Task 


INSTRUCTIONS  FOR  RANKING  TASK  1 

We  would  now  like  you  to  reassemble  the  cards  into  a  single  deck.  Then,  rank  the 
information  on  the  cards  according  to  importance  for  display  at  CONN. 

For  context,  the  operational  environment  should  be  assumed  to  be  a  SSN  in  the  following 
scenario: 

Mission:  ( littoral,  enemy  contacts) 

The  world  situation  is  that  a  war  has  broken  out  between  Russia  and  Ukraine  over  control  of  the 
former  Soviet  Navy.  Both  nations  are  trying  to  get  the  U.S.  involved  and  have  been  attacking 
U.S.  shipping  with  submarines  and  then  blaming  each  other.  You  are  patrolling  in  the  eastern 
Mediterranean  Sea.  Your  mission  is  to  protect  U.S.  shipping  lanes.  Your  orders  are  to  search  and 
destroy  any  enemy  submarines  in  the  vicinity  of  U.S.  shipping.  There  are  several  surface 
contacts  (i.e.,  merchants,  fishing  trawlers),  and  no  subsurface  contacts.  The  broad  band  range  of 
the  day  for  a  subsurface  contact  is  3-6  Kyd.  Intelligence  has  indicated  a  Kilo-type  former  Soviet 
Submarine  is  patrolling  in  the  area.  There  are  no  friendly  submarines  in  your  patrol  area.  The 
month  is  April  and  the  sea  state  is  2. 

OS  Course:  120  Speed:  5  Operating  Depth:  250-400ft 


INSTRUCTIONS  FOR  RANKING  TASK  2 

Again,  we  would  like  you  to  reassemble  the  cards  into  a  single  deck.  Then,  rank  the 
information  on  the  cards  according  to  importance  for  display  at  CONN. 

For  context  and  unlike  Ranking  Task  1,  the  operational  environment  should  be  assumed 
to  be  a  SSN  in  the  following  scenario: 

Mission:  (littoral,  no  enemy  contacts) 

The  world  situation  is  that  it  is  peacetime  and  there  are  no  conflicts  brewing.  You  are  on  a 
training  exercise  in  the  eastern  Mediterranean  Sea.  There  are  several  surface  contacts  (i.e., 
merchants,  fishing  trawlers)  and  there  is  a  friendly  submarine  in  your  patrol  area,  a  Trafalgar 
class  British  sub.  The  broad  band  range  of  the  day  for  a  subsurface  contact  is  3-6  Kyd.  The 
month  is  April  and  the  sea  state  is  2. 

OS  Course:  120  Speed:  5  Operating  Depth:  250-400ft 


37 


INSTRUCTIONS  FOR  RANKING  TASK  3 


We  would  now  like  you  to  reassemble  the  cards  into  a  single  deck.  Then,  rank  the 
information  on  the  cards  according  to  importance  for  display  at  CONN. 

Unlike  the  previous  scenarios,  the  operational  environment  should  be  assumed  to  be  a 
SSN  in  the  following  scenario: 

Mission:  ( deep  water,  enemy  contacts ) 

The  world  situation  is  that  a  war  has  broken  out  between  Russia  and  Ukraine  over  control  of  the 
former  Soviet  Navy.  Both  nations  are  trying  to  get  the  U.S.  involved  and  have  been  attacking 
U.S.  shipping  with  submarines  and  then  blaming  each  other.  You  are  underway  from  the  East 
Coast  and  are  cruising  along  to  the  eastern  Mediterranean  Sea.  Once  you  arrive,  your  mission  is 
to  protect  U.S.  shipping  lanes.  Your  orders  are  to  search  and  destroy  any  enemy  submarines  in 
the  vicinity  of  U.S.  shipping.  While  you  are  underway,  in  the  middle  of  the  Atlantic  Ocean, 
intelligence  has  indicated  a  Kilo-type  former  Soviet  Submarine  is  in  the  area.  The  month  is  April 
and  the  sea  state  is  2. 

OS  Course:  120  Speed:  20  Operating  Depth:  500 


INSTRUCTIONS  FOR  RANKING  TASK  4 

We  would  now  like  you  to  reassemble  the  cards  into  a  single  deck.  Then,  rank  the 
information  on  the  cards  according  to  importance  for  display  at  CONN. 

Unlike  the  previous  scenarios,  the  operational  environment  should  be  assumed  to  be  a 
SSN  in  the  following  scenario: 

Mission:  ( deep  water,  no  enemy  contacts) 

The  world  situation  is  that  it  is  peacetime  and  there  are  no  conflicts  brewing.  You  are  underway 
in  the  Atlantic  Ocean  to  a  training  exercise  in  the  eastern  Mediterranean  Sea.  There  are  a  few 
commercial  surface  contacts  and  there  is  a  friendly  submarine  in  the  area,  a  Trafalgar  class 
British  sub.  The  broad  band  range  of  the  day  for  a  subsurface  contact  is  3-6  Kyd.  The  month  is 
April  and  the  sea  state  is  2. 

OS  Course:  120  Speed:  20  Operating  Depth:  500 
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APPENDIX  D  MDS  Coordinates  for  the  20  Information  Items  for  All  Subject  Groups 

Table  1.  Coordinates  of  Information  Items  in  Two-Dimensional  Solution  for  SOBCpre  subjects 


Information  Item 

Dimension  1 

Dimension  2 

ESM 

-0.2 

-0.78 

OS 

-0.91 

0.08 

PMFL 

-0.5 

0.73 

OE 

-0.77 

-0.21 

CD 

-0.65 

0.04 

PROP 

-0.27 

1.16 

SSRCHPLN 

0.86 

-0.12 

VIS 

-0.39 

-0.75 

POT 

0.16 

1.44 

ACT 

0.85 

0.08 

ATM 

0.07 

1.41 

SAUD 

0.85 

-0.43 

SAN 

0.13 

1.33 

GEO 

-0.5 

-0.59 

TOS 

-0.95 

-0.42 

NTFCS 

-0.14 

-1.01 

SDIS 

0.68 

-0.51 

SLINUP 

0.88 

0.04 

TFCS 

-0.08 

-1.07 

SCLASDIS 

0.86 

-0.41 

Table  2.  Coordinates  of  Information  Items  in  Two-Dimensional  Solution  for  SOBCpst  subjects 


Information  Item 

Dimension  1 

Dimension  2 

ESM 

0.73 

0.63 

OS 

-0.31 

0.52 

PMFL 

-0.69 

-0.20 

OE 

0.08 

-0.55 

CD 

-0.28 

0.36 

PROP 

-1.32 

0.13 

SSRCHPLN 

0.60 

-0.72 

VIS 

0.67 

0.64 

POT 

-1.44 

-0.28 

ACT 

0.67 

-0.72 

ATM 

-1.43 

-0.22 

SAUD 

0.70 

-0.67 

SAN 

-1.45 

-0.27 

GEO 

0.32 

0.85 

TOS 

0.06 

0.78 

NTFCS 

0.46 

0.78 

SDIS 

0.74 

-0.50 

SLINUP 

0.69 

-0.75 

TFCS 

0.49 

0.80 

SCLASDIS 

0.71 

-0.61 
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Table  3.  Coordinates  of  Information  Items  in  Two-Dimensional  Solution  for  SOAC  subjects 


Information  Item 

Dimension  1 

Dimension  2 

ESM 

0.61 

0.61 

OS 

-0.25 

0.56 

PMFL 

-0.77 

-0.18 

OE 

0.04 

-0.50 

CD 

-0.12 

0.04 

PROP 

-1.24 

0.22 

SSRCHPLN 

0.40 

-0.86 

VIS 

0.62 

0.63 

POT 

-1.36 

-0.20 

ACT 

0.46 

-0.86 

ATM 

-1.36 

-0.20 

SAUD 

0.79 

-0.65 

SAN 

-1.36 

-0.20 

GEO 

0.46 

0.80 

TOS 

0.22 

0.88 

NTRGTFCS 

0.45 

0.99 

SDIS 

0.74 

-0.48 

SLINUP 

0.38 

-1.07 

TFCS 

0.55 

0.99 

SCLASDIS 

0.72 

-0.54 

Table  4.  Coordinates  of  Information  Items  in  Two-Dimensional  Solution  for  Post  DH  subjects 


Information  Item 

Dimension  1 

Dimension  2 

ESM 

-0.43 

0.45 

OS 

-0.41 

0.43 

PMFL 

0.17 

-0.77 

OE 

0.1 

-0.73 

CD 

0.05 

-0.71 

PROP 

0.89 

0.05 

SSRCHPLN 

0.02 

-0.69 

VIS 

-0.52 

0.57 

POT 

1.08 

0.1 

ACT 

0.07 

-0.7 

ATM 

1 

0.07 

SAUD 

0.49 

1.03 

SAN 

1.16 

0.1 

GEO 

-0.72 

0.69 

TOS 

-0.51 

0.58 

NTFCS 

-0.55 

0.59 

SDIS 

-0.93 

0.73 

SLINUP 

0.26 

-1.03 

TFCS 

-1.2 

1.03 

SCLASDIS 

0 

-1.76 
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APPENDIX  E  Ranking  Data 

Table  1.  Ranking  data  for  SOBCpre  participants 


Rank 

Deep/Neutral 

Deep/Hostile 

Shallow/Neutral 

Shallow/Hostile 

1 

Own  Ship  Data 

Own  Ship  Data 

Own  Ship  Data 

Own  Ship  Data 

2 

Geosit/Ops  Summary 

Geosit/Ops  Summary 

Target  Fire  Control  Solution 

Geosit/Ops  Summary 

3 

Target  Fire  Control  Solution 

Ocean  Environment 

Geosit/Ops  Summary 

Ocean  Environment 

4 

Counterdetectability 

Sonar  Detection  Displays 

Counterdetectability 

Sonar  Detection  Displays 

5 

Ocean  Environment 

Counterdetectability 

Ocean  Environment 

Propulsion  Plant  Lineup 

6 

Sonar  Detection  Displays 

NonTarget  Fire  Control  Solution 

Sonar  Detection  Displays 

Counterdetectability 

7 

Trial  Own  Ship 

Visual  Contact  Data 

Trial  Own  Ship 

NonTarget  Fire  Control  Solution 

8 

NonTarget  Fire  Control  Solution 

Propulsion  Plant  Lineup 

Propulsion  Plant  Lineup 

Visual  Contact  Data 

9 

Propulsion  Plant  Lineup 

Target  Fire  Control  Solution 

NonTarget  Fire  Control  Solution 

Trial  Own  Ship 

10 

ESM  Contact  Data 

ESM  Contact  Data 

Sonar  Search  Plan 

Target  Fire  Control  Solution 

11 

Sonar  Search  Plan 

Trial  Own  Ship 

ESM  Contact  Data 

ESM  Contact  Data 

12 

Visual  Contact  Data 

Sonar  Search  Plan 

Visual  Contact  Data 

Sonar  lineup 

13 

Sonar  lineup 

Sonar  lineup 

Sonar  lineup 

Sonar  Search  Plan 

14 

Sonar  Class  Displays 

Sonar  Class  Displays 

Sonar  Tracker 

Sonar  Class  Displays 

15 

Sonar  Tracker 

Sonar  Tracker 

Sonar  Class  Displays 

Sonar  Tracker 

16 

Active  Sonar  Lineup 

Active  Sonar  Lineup 

Active  Sonar  Lineup 

Active  Sonar  Lineup 

17 

PMFL  Data 

Ship's  Atmosphere 

Ship's  Atmosphere 

Ship's  Atmosphere 

18 

Ship's  Atmosphere 

PMFL  Data 

PMFL  Data 

PMFL  Data 

19 

Potable  Water  Status 

Potable  Water  Status 

Potable  Water  Status 

Potable  Water  Status 

20 

Sanitary  Tank  Status 

Sanitary  Tank  Status 

Sanitary  Tank  Status 

Sanitary  Tank  Status 
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Table  2.  Ranking  data  for  SOBCpst  participants 


Rank 

Deep/Neutral 

Deep/Hostile 

Shallow/Neutral 

Shallow/Hostile 

1 

Own  Ship  Data 

Own  Ship  Data 

Own  Ship  Data 

Own  Ship  Data 

2 

Counterdetectability 

Geosit/Ops  Summary 

Counterdetectability 

Geosit/Ops  Summary 

3 

Target  Fire  Control  Solution 

Trial  Own  Ship 

Target  Lire  Control  Solution 

Counterdetectability 

4 

Geosit/Ops  Summary 

Counterdetectability 

Geosit/Ops  Summary 

NonTarget  Lire  Control  Solution 

5 

Sonar  Detection  Displays 

NonTarget  Lire  Control  Solution 

Ocean  Environment 

Sonar  Lineup 

6 

Ocean  Environment 

Sonar-  Detection  Displays 

Sonar  Detection  Displays 

Trial  Own  Ship 

7 

Trial  Own  Ship 

Ocean  Environment 

Sonar-  Lineup 

Ocean  Environment 

8 

Sonar  Lineup 

Sonar  Search  Plan 

Trial  Own  Ship 

Sonar  Search  Plan 

9 

Propulsion  Plant  Lineup 

Sonar-  Lineup 

NonTarget  Lire  Control  Solution 

Propulsion  Plant  Lineup 

10 

NonTarget  Lire  Control  Solution 

Target  Lire  Control  Solution 

Propulsion  Plant  Lineup 

Sanitary  Tank  Status 

11 

Sonar  Search  Plan 

Propulsion  Plant  Lineup 

Sonar  Search  Plan 

Sonar  Detection  Displays 

12 

ESM  Contact  Data 

ESM  Contact  Data 

ESM  Contact  Data 

ESM  Contact  Data 

13 

Sanitary  Tank  Status 

Visual  Contact  Data 

Sanitary  Tank  Status 

Target  Lire  Control  Solution 

14 

Visual  Contact  Data 

Sanitary  Tank  Status 

Active  Sonar  Lineup 

Sonar  Class  Displays 

15 

Active  Sonar-  Lineup 

Active  Sonar  Lineup 

Visual  Contact  Data 

PMLL  Data 

16 

PMLL  Data 

Ship's  Atmosphere 

Ship's  Atmosphere 

Active  Sonar  Lineup 

17 

Ship's  Atmosphere 

PMLL  Data 

PMLL  Data 

Visual  Contact  Data 

18 

Sonar  Tracker/Cursor 

Sonar-  Tracker/Cursor 

Sonar  Tracker/Cursor 

Ship's  Atmosphere 

19 

Sonar  Class  Displays 

Sonar  Class  Displays 

Sonar  Class  Displays 

Potable  Water  Status 

20 

Potable  Water  Status 

Potable  Water  Status 

Potable  Water  Status 

Sonar  Tracker/Cursor 
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Table  3.  Ranking  data  for  SO  AC  participants 


Rank 

Deep/Neutral 

Deep/Hostile 

Shallow/Neutral 

Shallow/Hostile 

1 

Sonar  Detection  Displays 

Own  Ship  Data 

Sonar  Detection  Displays 

Sonar  Detection  Displays 

2 

Own  Ship  Data 

Sonar  Detection  Displays 

Own  Ship  Data 

Own  Ship  Data 

3 

Target  Fire  Control  Solution 

Geosit/Ops  Summary 

Target  Lire  Control  Solution 

Geosit/Ops  Summary 

4 

Geosit/Ops  Summary 

Target  Lire  Control  Solution 

Geosit/Ops  Summary 

Target  Lire  Control  Solution 

5 

Visual  Contact  Data 

Visual  Contact  Data 

NonTarget  Lire  Control  Solution 

Trial  Own  Ship 

6 

NonTarget  Fire  Control  Solution 

NonTarget  Lire  Control  Solution 

Trial  Own  Ship 

NonTarget  Lire  Control  Solution 

7 

Trial  Own  Ship 

Trial  Own  Ship 

Ocean  Environment 

Visual  Contact  Data 

8 

ESM  Contact  Data 

ESM  Contact  Data 

Visual  Contact  Data 

ESM  Contact  Data 

9 

Ocean  Environment 

Ocean  Environment 

Sonar  Class  Displays 

Ocean  Environment 

10 

Sonar  Class  Displays 

Sonar  Class  Displays 

Counterdetectability 

Sonar  Search  Plan 

11 

Counterdetectability 

Counterdetectability 

ESM  Contact  Data 

Sonar  Class  Displays 

12 

Sonar  Search  Plan 

Propulsion  Plant  Lineup 

Sonar  Search  Plan 

Counterdetectability 

13 

Sonar  Tracker/Cursor 

Sonar  Search  Plan 

Propulsion  Plant  Lineup 

Propulsion  Plant  Lineup 

14 

Propulsion  Plant  Lineup 

Sonar  Tracker/Cursor 

Sonar  Tracker/Cursor 

Sonar  Tracker/Cursor 

15 

Sonar  Lineup 

Sonar  Lineup 

Sonar  Lineup 

Sonar  Lineup 

16 

Active  Sonar  Lineup 

Ship's  Atmosphere 

Active  Sonar  Lineup 

Ship's  Atmosphere 

17 

Ship's  Atmosphere 

Active  Sonar  Lineup 

Ship's  Atmosphere 

Active  Sonar  Lineup 

18 

Sanitary  Tank  Status 

Sanitary  Tank  Status 

Sanitary  Tank  Status 

Sanitary  Tank  Status 

19 

PMLL  Data 

Potable  Water  Status 

Potable  Water  Status 

Potable  Water  Status 

20 

Potable  Water  Status 

PMLL  Data 

PMLL  Data 

PMLL  Data 
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Table  4.  Ranking  data  for  Post  DH  participants 


Rank 

Deep/Neutral 

Deep/Hostile 

Shallow/Neutral 

Shallow/Hostile 

1 

Sonar  Detection  Displays 

Sonar  Detection  Displays 

Sonar  Detection  Displays 

Sonar  Detection  Displays 

2 

Target  Fire  Control  Solution 

Own  Ship  Data 

Own  Ship  Data 

Own  Ship  Data 

3 

Own  Ship  Data 

Geosit/Ops  Summary 

Geosit/Ops  Summary 

Geosit/Ops  Summary 

4 

Geosit/Ops  Summary 

NonTarget  Lire  Control  Solution  NonTarget  Lire  Control  Solution 

Sonar  Class  Displays 

5 

NonTarget  Fire  Control  Solution 

Sonar  Class  Displays 

Target  Lire  Control  Solution 

NonTarget  Lire  Control  Solution 

6 

Sonar  Class  Displays 

Counterdetectability  Data 

Counterdetectability  Data 

Ocean  Environment 

7 

Trial  Own  Ship 

Ocean  Environment 

Trial  Own  Ship 

Trial  Own  Ship 

8 

Ocean  Environment 

Sonar  Tracker/Cursor  Audio 

Sonar  Class  Displays 

Counterdetectability  Data 

9 

Sonar  Search  Plan 

Trial  Own  Ship 

Ocean  Environment 

Propulsion  Plant  Lineup 

10 

Sonar  Lineup 

Propulsion  Plant  Lineup 

Propulsion  Plant  Lineup 

Sonar  Search  Plan 

11 

Counterdetectability  Data 

Sonar  Lineup 

Sonar  Lineup 

Sonar  Lineup 

12 

ESM  Contact  Data 

Target  Lire  Control  Solution 

Sonar  Search  Plan 

Sonar  Tracker/Cursor  Audio 

13 

Sonar  Tracker/Cursor  Audio 

Ship's  Atmosphere 

Sonar  Tracker/Cursor  Audio 

Target  Lire  Control  Solution 

14 

Active  Sonar  Lineup 

Sonar  Search  Plan 

Active  Sonar  Lineup 

Ship's  Atmosphere 

15 

Propulsion  Plant  Lineup 

Visual  Contact  Data 

PMLL  Data 

Potable  Water  Status 

16 

Ship's  Atmosphere 

Potable  Water  Status 

Ship's  Atmosphere 

PMLL  Data 

17 

PMLL  Data 

Sanitary  Tank  Status 

ESM  Contact  Data 

ESM  Contact  Data 

18 

Visual  Contact  Data 

ESM  Contact  Data 

Potable  Water  Status 

Visual  Contact  Data 

19 

Potable  Water  Status 

PMLL  Data 

Visual  Contact  Data 

Active  Sonar  Lineup 

20 

Sanitary  Tank  Status 

Active  Sonar  Lineup 

Sanitary  Tank  Status 

Sanitary  Tank  Status 
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The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 
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of  information,  including  suggestions  for  reducing  the  burden,  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports 
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14.  Abstract  (cont) 

Eighty-three  Naval  Officers,  some  qualified  and  some  not  qualified  as  OOD,  performed 
information  sorting  and  ranking  tasks  to  indicate  similarity  and  relative  importance  among  20 
categories  of  information  available  on  submarines.  These  20  submarine  concepts  were  primary 
elements  of  information  the  submarine  OOD  would  encounter  while  on  watch,  with  the 
exception  of  tactics  information.  Each  of  the  20  concepts  was  printed  on  an  index  card.  Research 
participants  made  similarity  judgments  by  sorting  the  20  cards  into  discrete  groups,  according  to 
the  participant's  own  definition  of  similarity.  The  number  of  piles  and  the  concepts  in  each  pile 
were  recorded.  Then,  participants  placed  the  20  cards  into  a  single  pile,  ranked  or  ordered 
according  to  relative  importance.  Participants  repeated  the  ranking  task  four  times  for  four 
differing  operational  scenarios.  Research  participants  were  Submarine  Officer  Basic  Course 
students  (both  pre  and  post  course),  Submarine  Officer  Advanced  Course  students,  and  Post 
Department  Head  submariners. 

The  results  show  that  more  experienced  personnel  possess  different  mental  models  than  less 
experienced  personnel.  With  regard  to  information  similarity,  submarine  OODs  organize 
information  in  two  basic  dimensions,  one  defined  by  the  source  of  information  (context/noise  v. 
contact/signal),  and  the  other  defined  by  the  destination  (or  primary  user)  of  the  information 
(sonar  v.  CONN).  In  addition  to  the  two-dimensional  structure  in  the  organization  of  all  20 
information  items,  there  was  also  a  more  finely  grained  grouping  or  "clustering"  of  information. 
The  expert  group  organized  information  into  four  clusters  and  the  groups  with  less  expertise 
organized  information  into  five  clusters.  Further,  the  information  in  these  five  clusters  varied  as  a 
function  of  experience.  With  regard  to  information  prioritization,  the  information  regarded  as  the 
most  important  varied  as  a  function  of  operational  scenario  (i.e.,  littoral  v.  pelagic  water 
environment;  neutral  v.  hostile  contact).  This  change  in  prioritization  between  scenarios  was 
most  evident  for  the  most  experienced  submariner  participants;  little  change  was  observed  for  the 
least  experienced. 

This  knowledge  allows  system  designers  to  develop  command  and  control  displays  that  use  this 
cognitive  organization  to  the  user's  advantage  by  developing  systems  that  are  intuitive  to  the 
user.  Additionally,  the  knowledge  of  these  mental  models  of  experts  could  be  used  as  targets  for 
training  less  experienced  personnel. 


Standard  Form  298 
Page  2 


